{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pima Indians Diabetes Data Set——Logistic回归\n",
    "数据说明： Pima Indians Diabetes Data Set（皮马印第安人糖尿病数据集） 根据现有的医疗信息预测5年内皮马印第安人糖尿病发作的概率。\n",
    "\n",
    "数据集共9个字段: 0列为pregnants(怀孕次数)； 1列为Plasma_glucose_concentration(口服葡萄糖耐量试验中2小时后的血浆葡萄糖浓度)； 2列为blood_pressure(舒张压,单位:mm Hg） 3列为Triceps_skin_fold_thickness(三头肌皮褶厚度,单位：mm） 4列为serum_insulin(餐后血清胰岛素,单位:mm） 5列为BMI,体重指数（体重（公斤）/ 身高（米）^2） 6列为Diabetes_pedigree_function(糖尿病家系作用) 7列为Age(年龄) 8列为Target(分类变量,0或1）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#导入必要的工具包\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "train1 = pd.read_csv('FE_pima_indians_diabetes.csv')\n",
    "train2 = pd.read_csv('FE_diabetes_tfidf.csv')\n",
    "\n",
    "#去掉多余的列\n",
    "train2 = train2.drop(['Target'], axis = 1)\n",
    "train = pd.concat([train1, train2], axis = 1, ignore_index = False)\n",
    "train.head()\n",
    "\n",
    "del train1\n",
    "del train2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 768 entries, 0 to 767\n",
      "Data columns (total 17 columns):\n",
      "pregnants                             768 non-null float64\n",
      "Plasma_glucose_concentration          768 non-null float64\n",
      "blood_pressure                        768 non-null float64\n",
      "Triceps_skin_fold_thickness           768 non-null float64\n",
      "serum_insulin                         768 non-null float64\n",
      "BMI                                   768 non-null float64\n",
      "Diabetes_pedigree_function            768 non-null float64\n",
      "Age                                   768 non-null float64\n",
      "Target                                768 non-null int64\n",
      "pregnants_tfidf                       768 non-null float64\n",
      "Plasma_glucose_concentration_tfidf    768 non-null float64\n",
      "blood_pressure_tfidf                  768 non-null float64\n",
      "Triceps_skin_fold_thickness_tfidf     768 non-null float64\n",
      "serum_insulin_tfidf                   768 non-null float64\n",
      "BMI_tfidf                             768 non-null float64\n",
      "Diabetes_pedigree_function_tfidf      768 non-null float64\n",
      "Age_tfidf                             768 non-null float64\n",
      "dtypes: float64(16), int64(1)\n",
      "memory usage: 102.1 KB\n"
     ]
    }
   ],
   "source": [
    "train.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = train['Target']\n",
    "X_train = train.drop(['Target'], axis = 1)\n",
    "\n",
    "#保存特征名字以备后用（可视化）\n",
    "feat_names = X_train.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 默认参数的Logistics Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "lr = LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logloss of each fold is: [0.48709996 0.52700043 0.45879341 0.4159791  0.48141637]\n",
      "cv logloss is: 0.4740578530355644\n"
     ]
    }
   ],
   "source": [
    "#交叉验证用于评估模型性能和进行参数调优（模型选择）\n",
    "#采样5折交叉验证\n",
    "from sklearn.model_selection import cross_val_score\n",
    "loss = cross_val_score(lr, X_train, y_train, cv = 5, scoring = 'neg_log_loss')\n",
    "print('logloss of each fold is:', -loss)\n",
    "print('cv logloss is:', -loss.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "原始特征：0.47615970944434044\n",
    "log特征：0.581196363512364\n",
    "tfidf特征：0.6162589981975196\n",
    "原始特征 + tfidf特征:0.4740578530355644"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 正则化的 Logistic Regression及参数调优\n",
    "logistic回归的需要调整超参数有：C（正则系数，一般在log域（取log后的值）均匀设置候选参数）和正则函数penalty（L2/L1） 目标函数为：J = C sum(logloss(f(xi), yi)) + penalty"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise',\n",
       "       estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False),\n",
       "       fit_params=None, iid=True, n_jobs=4,\n",
       "       param_grid={'penalty': ['l1', 'l2'], 'C': [0.1, 1, 10, 100, 1000]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#导入模块\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "#设置参数搜索范围\n",
    "penaltys = ['l1', 'l2']\n",
    "Cs = [0.1, 1, 10, 100, 1000]\n",
    "tuned_parameters = dict(penalty = penaltys, C = Cs)\n",
    "\n",
    "#生成学习器实例\n",
    "lr_penalty = LogisticRegression(solver = 'liblinear')\n",
    "\n",
    "#生成GridSearchCV的实例\n",
    "grid = GridSearchCV(lr_penalty, tuned_parameters, cv = 5, scoring = 'neg_log_loss', n_jobs = 4)\n",
    "\n",
    "#调用GridSearchCV的fit方法\n",
    "grid.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4737118897870254\n",
      "{'C': 1, 'penalty': 'l1'}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid.best_score_)\n",
    "print(grid.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "d:\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xt8VPWd8PHPdyYzuUMCCXIJyC1U7gIBbYuXWi9oW1GL1brU1W3X2qc8ttvLSnetu0X32Vbd9dkL+7SsitraesFaoiJsq8VLWy6xooCoBCo1BCUGCOY+l+/zxzkzmRmSzARykhC+79frOOf8zu+c+c5Iznd+53fO74iqYowxxnTH198BGGOMGfgsWRhjjEnLkoUxxpi0LFkYY4xJy5KFMcaYtCxZGGOMScuShTHGmLQsWRhjjEnLkoUxxpi0srzcuYgsAv4N8AP3qeoPU9bfANwN7HeL/lNV7xORM4H/BwwBIsA/qepj3b1XSUmJjh8/vnc/gDHGDHKvvvrqh6pamq6eZ8lCRPzASuAioAbYKiKVqvpmStXHVHVZSlkzcL2q7haR0cCrIrJBVY909X7jx4+nqqqqNz+CMcYMeiKyL5N6Xp6GWgBUq+peVW0HHgUWZ7Khqr6jqrvd+VrgIJA28xljjPGGl8liDPBewnKNW5bq8yLyhoisEZGxqStFZAEQBPZ4E6Yxxph0vEwW0klZ6hC3TwPjVXUW8BvgoaQdiIwCfgrcqKrRY95A5CYRqRKRqrq6ul4K2xhjTCovO7hrgMSWQhlQm1hBVesTFv8b+FFsQUSGAM8Ct6nqps7eQFVXAasAKioqbKx1YwwAoVCImpoaWltb+zuUASMnJ4eysjICgcBxbe9lstgKlIvIBJyrna4FrkusICKjVPWAu3g5sMstDwJPAQ+r6hMexmiMGYRqamooLCxk/PjxiHR2kuPUoqrU19dTU1PDhAkTjmsfnp2GUtUwsAzYgJMEHlfVnSKyQkQud6vdIiI7ReR14BbgBrf8C8C5wA0iss2dzvQqVmPM4NLa2srw4cMtUbhEhOHDh59QS8vT+yxUdR2wLqXs9oT57wHf62S7nwE/8zI2Y8zgZoki2Yl+H3YHtzHGANf85A9c85M/9HcYA5YlC2PMgDJYDtoFBQXx+UWLFlFUVMRnP/vZeNmVV17JmWeeyeTJkxk6dChnnnkmZ555Jr///e979D4/++WzrHnuhV6LuyuenoYyxhgD3/3ud2lubuYnP/lJvOypp54CYOPGjdxzzz0888wzx7XvV155luJhxSy59IJeibUr1rIwxgwo7wbv4d3gPf0dRq/69Kc/TWFhYcb1t27dynnnnce8efO49NJL+eCDDwC49957mTZtGrNnz2bp0qXs2bOHNT97ggdW3n9crZKesJaFMR6LnVJ57Ksf7+dITk0/eHonb9YeTVvvzQNOnUxOgU0bPYR/+Nz0E46tM21tbXzjG9+gsrKSkpISHnnkEb7//e+zatUq7rrrLvbt20cwGOTIkSMUFRWxZOnVFA8r5p9v+ydP4omxZGGMMQPIrl272LlzJxdeeCEAkUiEsrIyAKZPn87SpUtZvHgxV1xxRZ/GZcnCGDOoZdoCGCgtQFVl1qxZvPzyy8es27BhAy+++CJr167lzjvvZMeOHX0Wl/VZGGPMADJt2jT279/Pli1bAGhvb2fnzp1EIhFqamq44IILuPvuu6mrq6O5uZn8gnyampo8j8uShTEeG4wdtqZnzjnnHK6++mqef/55ysrK2LBhQ5d1s7OzWbNmDd/61reYPXs2c+bMYfPmzYTDYa677jpmzZrF3LlzufXWWyksLOSCSy9k/drnmDNnjnVwG2PMyaaxsTE+39kppZjzzz+f888/P6ls7ty5vPLKK8fU/d3vfndM2cTySax96Vmmlkw+/mAzYMnCGGPo/76Kgc5OQxljjEnLkoUxxpi0LFkYY4xJy5KFMcaYtCxZGGMMwOrPOJPplKfJQkQWicjbIlItIss7WX+DiNQlPA3vKwnr1ovIERE5vqEYjTGmH/XFEOUrV67k6TVrezXurnh26ayI+IGVwEVADbBVRCpV9c2Uqo+p6rJOdnE3kAd81asYjTGmL5zIEOXhcJisrM4P1V//+tfZ9WF17wfcCS9bFguAalXdq6rtwKPA4kw3VtXngY+8Cs4cvxvX38iN62/s7zCMOWn0dIjysrIy7rjjDj75yU/y1FNP8eMf/5j58+cze/Zsrr76alpaWgC47bbbePjHqwFYuHAhy5cvZ8GCBXzsYx/r9bu5vbwpbwzwXsJyDXBWJ/U+LyLnAu8Af6Oq73VSp1MichNwE8C4ceNOIFRjzKD13HJ4f3v6eu+/4bxm0m8xciZc+sMTiyuN/Pz8+B3b9fX13HzzzQAsX76cBx98kK997WvHbKOqbNmyhcrKSlasWMH69et7LR4vWxadPR1cU5afBsar6izgN8BDPXkDVV2lqhWqWlFaWnqcYQ6exzgaYwaPa665Jj7/xhtvcM455zBz5kweffRRdu7c2ek2V111FQDz5s3j3Xff7dV4vGxZ1ABjE5bLgNrECqpan7D438CPPIzH9JLYQ2KMOSlk2gKItShufNa7WHogPz8/Pn/99dfz3HPPMWPGDO677z42bdrU6TbZ2dkA+P1+wuFwr8bjZctiK1AuIhNEJAhcC1QmVhCRUQmLlwO7PIzHGGNOSk1NTYwcOZJQKMTPf/7zfonBs5aFqoZFZBmwAfADD6jqThFZAVSpaiVwi4hcDoSBQ8ANse1F5GXgDKBARGqAL6tq1+P6GmPMAHXOOefw1ltv0djYSFlZGffffz+XXHJJxtuvWLGCBQsWMG7cOGbMmEFra6uH0XbO01FnVXUdsC6l7PaE+e8B3+ti23O8jM0YY7x0IkOU19TUJC0vW7aMZcuOvcPgzjvvjF86mzik+ciRI6mu7t1Lam2IcmOMgQHTVzFQ2XAfxhhj0rJkYYwxJi1LFsYYY9KyZGGMMSYt6+AG3g3e48492a9xGGP6T2y8s9WLVvdzJAOTtSyMMcYDsSHKt23bxsc//nGmT5/OrFmzeOyxx4DeGaIcYNPLf+D1qtd6Pf5U1rIwxhgP5eXl8fDDD1NeXk5tbS3z5s3jkksuyXiI8nQ2v/wHiocVc+2iq3sz7GNYy8IYYzw0ZcoUysvLARg9ejQjRoygrq6u2222bt3Keeedx7x587j00kv54IMPALj33nuZNm0as2fPZunSpezZs4c1P3uCB1bef1ytkp6wloUxZlD70ZYf8daht9LWi9XJ5FktZww7g1sX3NrjWLZs2UJ7ezuTJk3qsk5bWxvf+MY3qKyspKSkhEceeYTvf//7rFq1irvuuot9+/YRDAY5cuQIRUVFLFl6NcXDivnn2/6px/H0hCULY4zpAwcOHOBLX/oSDz30ED5f1yd1du3axc6dO7nwwgsBiEQilJWVATB9+nSWLl3K4sWLueKKK/ok7hhLFsaYQS3TFoCXV0MdPXqUz3zmM9x5552cffbZ3dZVVWbNmtXpeFIbNmzgxRdfZO3atdx5553s2LGj12PtivVZGGOMh9rb27nyyiu5/vrrufrq9J3Q06ZNY//+/WzZsiW+/c6dO4lEItTU1HDBBRdw9913U1dXR3NzM/kF+TQ1NXn9MSxZGGOMlx5//HFeeuklHnzwwfjlsdu2beuyfnZ2NmvWrOFb3/oWs2fPZs6cOWzevJlwOMx1113HrFmzmDt3LrfeeiuFhYVccOmFrF/7HHPmzLEObmOMOdnEhihfunQpS5cu7bJeZ0OUz507N2nI8ZjYM7kTTSyfxNqXnmVqyeQTCzgNT1sWIrJIRN4WkWoRWd7J+htEpE5EtrnTVxLW/aWI7Hanv/QyTmOMWb1otd293Q3PWhYi4gdWAhfhPI97q4hUquqbKVUfU9VlKdsOA/4BqAAUeNXd9rBX8RpjjOmaly2LBUC1qu5V1XbgUWBxhtteAvxaVQ+5CeLXwCKP4jTGGJOGl8liDPBewnKNW5bq8yLyhoisEZGxPdzWGGNMH/AyWUgnZZqy/DQwXlVnAb8BHurBtojITSJSJSJV6W6fN8YYc/y8TBY1wNiE5TKgNrGCqtarapu7+N/AvEy3dbdfpaoVqlpRWlraa4EbY049+750Pfu+dH1/hzFgeZkstgLlIjJBRILAtUBlYgURGZWweDmwy53fAFwsIsUiUgxc7JYZY8xJoS+GKF+5ciVPr1nrSfypPLsaSlXDIrIM5yDvBx5Q1Z0isgKoUtVK4BYRuRwIA4eAG9xtD4nIHTgJB2CFqh7yKlZjjPHKiQ5RHg6Hycrq/FD99a9/nV0fVnsWeyJPb8pT1XXAupSy2xPmvwd8r4ttHwAe8DI+Y4zx2pQpU+LziUOUFxUVdblNWVkZX/3qV1m/fj3f/OY3qa+v5/7776e9vZ0pU6bw8MMPk5uby2233UYkR7n+5htZuHAhCxcu5IUXXqChoYHVq1fziU98otc+h93BbYwZ1N7/P/+Htl3phyhvfcupk0m/RfbUMxj5d3/X41gyGaI8Jj8/P37Hdn19PTfffDMAy5cv58EHH+RrX/vaMduoKlu2bKGyspIVK1awfv36HsfYFUsWxhjTBzIdojzmmmuuic+/8cYb3H777Rw5coSPPvqIz372s51uc9VVVwEwb9483n333V6JO8aShTFmUMu0BRBrUZz+04d7PYaeDFEek5+fH5+//vrree6555gxYwb33XcfmzZt6nSb7OxsAPx+P+Fw+MQDT2CjzhpjjId6OkR5Z5qamhg5ciShUIif//znvRxhZqxlYYwxHooNUV5fX8+DDz4IEB+uPFMrVqxgwYIFjBs3jhkzZtDa2upRtF2zZGGMMR44kSHKa2pqkpaXLVvGsmVJ460CcOedd8YvnU0c0nzkyJFUV/fuJbWWLIwxBm/6KgYT67MwxhiTliULY8ygpHrM2KOntBP9PixZGGMGnZycHOrr6y1huFSV+vp6cnJyjnsf1mdhjBl0ysrKqKmp4VR4dMGBRvcz1oW6rZeTk0NZWdlxv48lC2PMoBMIBJgwYUJ/h9Enblh9GwCbb3zS0/ex01DGGGPSsmRhjDEmLUsWxhhj0rJkYYwxJi1Pk4WILBKRt0WkWkSWd1NviYioiFS4y0ERWS0i20XkdRE538s4jTHGdM+zq6FExA+sBC4CaoCtIlKpqm+m1CsEbgE2JxT/NYCqzhSREcBzIjJfVaNexWuMMaZrXrYsFgDVqrpXVduBR4HFndS7A7gLSBxGcRrwPICqHgSOABUexmqMMaYbXiaLMcB7Ccs1blmciMwBxqpq6pPKXwcWi0iWiEwA5gFjU99ARG4SkSoRqToVbr4xxpj+4uVNedJJWfzeexHxAfcCN3RS7wFgKlAF7AN+Dxzz2CdVXQWsAqioqLD7+o0xxiNeJosaklsDZUBtwnIhMAPYKCIAI4FKEblcVauAv4lVFJHfA7s9jNUYY0w3vDwNtRUoF5EJIhIErgUqYytVtUFVS1R1vKqOBzYBl6tqlYjkiUg+gIhcBIRTO8aNMcb0Hc9aFqoaFpFlwAbADzygqjtFZAVQpaqV3Ww+AtggIlFgP/Alr+I0xhiTnqcDCarqOmBdStntXdQ9P2H+XeBjXsZmjDEmc3YHtzHGmLQsWRhjjEnLkoUxxpi0LFkYY4xJy5KFMcaYtCxZGGOMScuShTHGmLQsWRhjjEnLkoUxxpi0LFkYY4xJy5KFMcaYtHqcLETEJyJDvAjGGGPMwJRRshCRn4vIEHfY8DeBt0Xku96GZowxZqDItGUxTVWPAlfgjCI7Dhs23BhjThmZJouAiARwksVaVQ2R8IhUY4wxg1umyeInwLtAPvCSiJwOHE23kYgsEpG3RaRaRJZ3U2+JiKiIVLjLARF5SES2i8guEflehnEaY4zxQEbJQlX/XVXHqOpl6tgHfKq7bUTED6wELgWmAV8UkWmd1CsEbgE2JxRfDWSr6kxgHvBVERmfSazGGGN6X6Yd3N9wO7hFRO4XkT8CF6TZbAFQrap7VbUdeBRY3Em9O4C7gNaEMgXyRSQLyAXayaAlY4wxxhuZnob6K7eD+2KgFLgR+GGabcYA7yUs17hlcSIyBxirqs+kbLsGaAIOAH8G7lHVQxnGaowxppdlmizEfb0MWK2qryeUpdsmUbxTXER8wL3AtzuptwCIAKOBCcC3RWTiMW8gcpOIVIlIVV1dXfpPYYwx5rhkmixeFZH/wUkWG9x+hmiabWqAsQnLZUBtwnIhMAPYKCLvAmcDlW4n93XAelUNqepB4HdAReobqOoqVa1Q1YrS0tIMP4oxxpieyjRZfBlYDsxX1WYgiHMqqjtbgXIRmSAiQeBaoDK2UlUbVLVEVcer6nhgE3C5qlbhnHq6wO0jycdJJG/15INlKhJVjn5wNs2Hp/Ly7jpqj7SgalcFG2NMoqxMKqlqVETKgOtEBOBFVX06zTZhEVkGbAD8wAOqulNEVgBVqlrZzeYrgdXADpzTWatV9Y1MYu2puo/aaD06GY0G+dL9WwDIC/qZWJrPpNICJpUWxOcnlOSTE/B7EYYxxgxoGSULEfkhMB94xC26RUQ+oard3v+gqutw7vhOLLu9i7rnJ8w34lw+67mRQ3OoOP0HhCIF3HrJWvbUNbpTE1XvHmbtto4zZyJQVpwbTyLOlM+kEQUMzw/iJlJjjBl0MkoWOH0VZ6pqFEBEHgJeAwbFzXIiEMxq5OOThvPxScOT1rW0R9j7oZM89hzsSCR/2FNPW7ij22ZobsBJHKUFTBrRkUjGDcsjy2+D+54q2sNRPjjayv4jLdQeaWH/4RaOvv8JVP1894nX8Yng8wl+H/jdeZ8I/vgrTh23zO8TRJy6/qS6OPtJ2gcd24kg8X0k79OXUC4iSfv2JcTll4Sy1PdKKE/eN/ajaZDKNFkAFAGxy1eHehDLgJQb9DN99FCmj07+yNGosv9IC3s/TEwijWx8p44nXq2J1wv4hdOH53ckEjeZTCzNZ0hOoK8/jjkBqkpDS8hNBK3UxhKCO9UeaeHgR22kdnmJ/3REwvyu+kMiqkSiEFUlqkokqkSjSkSVqFseUT1mHycTn9CRFCUh4SUlnISkFE9uTp36I1cAymX/9jLg/JiLSZpPuOAyuTyBCIoiKM7FmFGQ2JcbdcucSQQ0No+CqLOtKErU3a+zXkWTto1PEvsB2VEW20fHfBQFxF3Gfa/E+h2x0RFHYv2E2BsOj8bn8/4fTKbJ4p+B10Tktzj/L85lkLQqjpfPJ4wdlsfYYXmcNyX5SqyGlhB73RbInrpG9hxspPpgI8/vOkg42vE/dURhdlKfiNMiyWf00Fx8Pvt11tdCkSjvN7TGE4Dz2pEUao+00NQeSdom6PcxuiiH0UW5nFteyuiiXMYU5TK6KDdeft7PvgDA7298MuNYNJZIlHhScRKKU+as0/hrNIqbiNTZNp6InPKou21sP+ruI5JQ7uw3Sigapj3STijaTnskTCjSTnu0nVA0RCjaTigSmw8TVmc+7E4hDRGJthPWMGENEYmGiMTm41OYiIaIaogIYaIaIuS+RgmTU9gIonyQFQSi8QOkc+FJ6gEV1D1wdhxIUw/Cg1tgOETbvb8aNNMO7l+IyEacfgsBblXV970M7GQ2NDfAnHHFzBlXnFQeikT586FmtyXSFG+NPP16LUdbw/F6OQEfE0s6kkdiR7t1sB8fVeVoSzieBGob3BbB4VgiaOWDj1qP+UU/PD/I6KJcJpbms7C8hDFJySCX4fnB40rsqko4GnYOwpEQ7dF22mMH5UiIUDTkHrBDnZYnbpf6GtsudTleHtuHu5+wmxxi67UXxwj1i5+gP0iWL4ugL0jQHyTgC5DnvjrLOQT9QYK+IAF/gBfe3QQIl05aiE98+MQ5jRubF8R5FYnPpy4n1vHhvqbuK6FckI46bnls22P2nbDPruokxZC4764+S+K+E7fp6v0T4vzc4zchWd6f6u42WYjI3JSi2PmV0SIyWlX/6E1Yg1PA74sf+BOpKh82tnd0rh9sYu+HjWx77zDPvFEbP4CJwJiihA72ER2JpKTg1O5gT2wV1DY4B//EfoOuWgWjinIYU5TLwvISt1WQE28djBqaS27QSc7tkXYaQ400tjfSGDpMU6iGHYcbaTzYSFOoicaQ+9re6NRLWG6RPwHKuY+em3Tw7k1ZkkXAH4gfjGMH5ixfVvxAHPQHyc3Kdeq5B+bEA3jqcrzcnY8f0GN1U94rsTxWz+/r+Y+bs1Z/HoA7F97Zq9/RYOWjb05np2tZ/Es365T040OZDIgIpYXZlBZmc/bE5A721lCEP33YFE8isYSy5U+HaAl1HPyG5GQldKx3XKU1blgegZO8g11VOdoa7qSPoOMU0QdHW4mm/CAuzg8wqsjPmOEwe2IWxQVCYV6EvJwwOcEQ4mujOdxxoD8YamTv0UYa6zsO9LFEkMnBPUuyKAgWUBAooCBYQH4gnxF5I3invgYQLh5/cdKBuLMDeuKBOfHXeOKBOeBPPoAHfIH4L1ZjvNJtslDVbkeWNd7LCfiZOmoIU0clP8k2GlUOHG1N6lzfc7CJl96pY01CB3uWTxg3PO+YJDKptIChuQOjgz0Uca4gih38aw43896RI9Q0HObA0SMcbGqgJdyE+NvA14r42sjKaqcgN0xuToiCsSGKstoQXxtRaSGkLbRGmmkONfGehp0ByhrcqRNBXzB+cI8d6Efmj6SgKLksPp+ynB/IpyBYQNDXeesu9kv5trNv8+w7NMZrmd5ncVUnxQ3Adnc4jpNWa7iVer9zpcMjux7BJz784o+fU0ya8OHzuetJXucXv3sZYsdrd+s6rYPg97mv4sfn8yW9T1JcPomfPz83pYP9aGuIvXXJV2ntqWti49sHCUU6fn6XFGQnJY9Y/8iYouPrYI9qlJZwS9Iv8sZQI3VNDexvOMyBj45Q19TAoeaPONL2EY3tjTSHm2iPNjtJwN+G+FrB1+ZcPeIDikCKIK+zN/Tn4Avkk5104B5BYbAw+UAeKCA/2LGcuj7oD/b4sxpzqsn0aqgvAx8Hfusun48zPMcUEVmhqj/1ILY+0RRq4mCWcwD94ZZ0A+kOHIkdaZ0lpaQEl+sj63Q/k08XolEhHIVwBEJh2BNR3qyDyAfOXlFBxEdOIEBeIIu8QIC87AAFwQD52QECfj9tsh9FWbpuKY3tjTS0fURTqInWSHNGnaMaDeLTHAK+XPJz8xkRGMKQ7NEU5QyhJG8IIwqGUpwzpNNf9bGDfV4gj4BvYLSMjDkVZJososBUVXUOKSKnAf8POAt4CThpk0VxTjFT2pzzvfdf/yIRjRDVaMdElGjUeY1oJD6fWCeiEfdyxZRtU9bF67j7jGgERYlE3dcM95O6Lrb+mHI3zsT9d7VdWyhMU3uYxvZ2mtvCNIdCNLa1Ud/U7F477lyKGMiCsOaABtjxXitt7blotAiN5qCRbDSaQ25WPsNyCinJH8rIwiJGDxnK2KJhjC8ezqThwxg5JN8uDTbmJJNpshgfSxSug8AUVT0kIr17WUcf84kPv3u7TVFOUT9HM/C0hiK8W9+U1Ln+7FuvIxLmkjMqnCuHinPjVxKNGppLfnZP7vU0xpwMMv2rfllEngGecJeX4DyLOx844klkZkDICfg5Y+QQzhjZ0cG+abVzSeO/XvOV/grLGNPHMk0WXweuAhbi3JT3EPCkOrdU2hVTxhgzyGV6B7eKyCs4z8JWYIvaQx+MMeaUkdGdPCLyBWALzumnLwCbRWSJl4EZY4wZODI9DfX3OE/JOwggIqXAb4A1XgVmjDFm4Mh0jABfys139ZlsKyKLRORtEakWkeXd1FsiIuo+fxsR+QsR2ZYwRUXkzAxjNcYY08sybVmsF5ENwC/c5WtIeQJeKhHx4zwe9SKcAQi3ikilqr6ZUq8QuAXYHCtT1Udwn8onIjOBtaq6LcNYjTHG9LKMWhaq+l1gFTALmA2sUtVb02y2AKhW1b2q2g48CizupN4dwF1Aaxf7+SIdScoTy59v4K9/2dXbG2OMyfjuKVV9Esj86S0wBpwx3Fw1OHd8x4nIHGCsqj4jIt/pYj/X0HmSQURuAm4CGDduXA9CS3DoT8ydeohoFPj5NTDzavjYpRDMP779GWPMIJTueRYfQaeD/QjOFbVDOlmXWCdVfF8i4gPuBW7o5v3PAppVdUdn61V1FU6Lh4qKiuO7lLd4PK+8Vsro0hYmFm2Hd9ZDIM9JGDOWwORPQ1b2ce3aGGMGi3RDlBeewL5rgLEJy2VAbcJyITAD2OgO6zwSqBSRy1W1yq1zLR6fgkKEo01BjjYFmbhqB/z5D7BjDez8Fex4EnKGwtTLYeYSGH8OHMfDXIwx5mTn5SA+W4FyEZkA7Mc58F8XW6mqDUBJbNl9bOt3YonCbXlcjfO8777h88H4TzrTpXfB3o2wfQ3sfApe+ykUnAbTr3RaHGUVyU+JN8aYQcyzZKGqYRFZBmwA/MADqrpTRFYAVapamWYX5wI1qrrXqxi75Q9A+UXOFGqBdzY4LY6q1bD5x1A8HmZ83kkcp03rlxCNMaaveDo8qKquI+USW1W9vYu656csbwTO9iq2HgnkwvQrnKm1AXY94ySOV/4vvPwvMGKamzg+D8Mm9He0xhjT62ws6Z7KGQpz/sKZGuvgzV85p6peuMOZxlQ4/RvTr4TCkf0drTHG9ApLFieioBQW/LUzHfkz7Pil0+JYvxw2/J3TIT5zCUz9HOQW93e0veb00J7+DsEY08cyHe7DpFM0DhZ+E25+Bb6+Bc79LjS8B5X/G+4uh1980WmBtDf1d6TGGNNj1rLwQunH4FN/B+d/D2pfcy7B3fFLeHsdBPKdezhmLoFJn4asYH9Ha4wxaVmy8JIIjJnrTBfdAX/+vdO6ePNXzumqnCKYdrlz1/jpn7R7OIwxA5Yli77i88H4hc502d2w57ew/QnY/iT88WEoGAkzrnIuxR0zd0Dfw3HD4+3OzE39G4cxpu9YsugP/gBMudiZ2pudIUZ2PAlb74NN/wXFE5zLcGcugRFT+ztaY4yxZNHvgnlui+IqaDkCbz3jnKp65V/h5XtgxHQAJIEMAAAXaUlEQVSY6d7DUTy+v6M1xpyiLFkMJLlFMGepMzUedIYZ2b4Gnl/hTGULOu7hKBjR39EaY04hliwGqoIRcNZXnenwPveKqifhub917uOYcK7TvzH1c06SMcYYD9l9FieD4tPhnG/B134H/2sznPNtJ4FULoN7yuEX1zmJpL25vyM1xgxS1rIAUB3QVx8lGXEGXHAbfOrvYf8fnUtwd/wS3n7WuYfjjM84p6omfsru4TDG9JpTPllEGps4fT+EA8r+b3+H7PLJZE+eTHZ5OYGyMsQ/QO99EIGyec508Z2w73fuPRxrYfvjzvAi0xY7p6pO/6Rz6a4xxhynUz5ZEA5xtBCCIWjZto2jzz4bXyXZ2WRPmuQkkPJygpMnk1NeTtbo0chAaon4/E4fxoRz4bJ7YM8LTovjjcfh1QehcBRMv8q5qmr0wL6HwxgzMJ3yycJfVMThoc7Bc95vfkO0qYm2PXto211N2+7dtFVX07RpMw1rOx6/4cvLIzh5stsKKSe7vJzs8slkjRjR/0kkKwgfW+RM7U3w9nNOf8aWVbBpJQyb6LQ2Zi5xhiUxnlv+83edmRv7NQxjToinyUJEFgH/hvPwo/tU9Ydd1FsCPAHMT3hS3izgJ8AQIOqua/UizjbJjc/78vPJnTWL3FmzkupEjh6lrbraSSLVTiJpfPElGp78Zce2Q4bET2Flx5JJeTlZw4d7EXZ6wXwnKcxcAi2HYdfTzqmql++Bl+6C02Z23MNRNK5/YjTGnBQ8SxYi4gdWAhfhPI97q4hUquqbKfUKgVuAzQllWcDPgC+p6usiMhwIeRVrJvxDhpA3dy55c+cmlYcPH3ZaIG4rpG33bj5av54jDQ0d2xYXdySQKe7r5Mn4i/rwktfcYph7vTN99IFzD8eONfCbf3SmsWc5LY7pV9g9HL0sW1v6OwRjTpiXLYsFQHXssagi8iiwGHgzpd4dwF3AdxLKLgbeUNXXAVS13sM4T0hWcTFZCxaQv2BBvExVCdfV0V7d0Qpp211Nw9q1RJs6hijPKi09pj8kOHky/oICb4MuPA3OvtmZDv0p4R6O78L6W2HCeR3P4cgZ6m0sxpiTgpfJYgzwXsJyDXBWYgURmQOMVdVnRCQxWUwBVEQ2AKXAo6p6l4ex9ioRITBiBIERI8j/xCfi5apK+P3348kj1ho5/PgTaEvHr8+s0aPc1kd5R4tk0kR8eXm9H+ywCXDud5zpgzed1sb2NbD26/DM30D5xU7imLLIebysMR77h0d2OTPWx5ORvvq+vEwWnfX0anyliA+4F7ihk3pZwEJgPtAMPC8ir6rq80lvIHIT7tin48YN/HPuIkJg1CgCo0ZRcO658XKNRgnt35+UQNp276Z502a0vT22MYGysoT+EKdTPThhAr7s7N4J8LRpcNrtcMH3Yf+rTtLY+UtnvKpggXMPx4wliCiqdkWVMacSL5NFDTA2YbkMqE1YLgRmABvdK4hGApUicrm77Yuq+iGAiKwD5gJJyUJVVwGrACoqKpSTlPh8BMeOJTh2LIUXfCperuEw7X9+j7bqjgTSXl1N40svQTjsVPL7CY4bl5RAssvLCZ5+OhIIHGdAAmUVznTJP8G7rzjDqe+qhDce49MLfNQfyYbf/4fzzPFRs50BEY0xg5aXyWIrUC4iE4D9wLXAdbGVqtoAlMSWRWQj8B1VrRKRPcDfikge0A6ch9MKOaVIVhbZEyeQPXECXHxxvFzb22nfty+pFdK2ezcfPf88RKNOpUCA7PHjndZHwhVawXHjenajoc8PE89zps/8C1Q/T92//xXFQ9rhf25zA/XDadPdBDPfSSDDJ9uNgMYMIp4lC1UNi8gyYAPOpbMPqOpOEVkBVKlqZTfbHhaRf8VJOAqsU9Vnu6p/qpFg0G1FlCeVR9vaaN+710kg7ziJpOWN7Rxd91zHttnZBCdOTLhHxL1bffRoJN3BPSsbzriM198ZBsBlv9oINVWwv8p53b4Gqh5w6mYPdR7iVFbhJI+yCsgv6XrfxpgBzdP7LFR1HbAupez2Luqen7L8M5zLZ02GfNnZ5EydSs7U5AcmRZuaaNu7N6lPpHlrFUcrn47Xkbw85271lNNZWaed1vWNhgUj4IzLnAmcVs2H73Qkj/1V8PK/gLqtnaLTE5LHfBg5EwI5XnwVxphedsrfwX0q8OXnkztzJrkzZyaVRz76qOM0VuxGw1depuGppzq2LSyM3xcSSyD+iBLprBHi8zkDHY44w3kmBzh3kddu60ggf97kXKYL4As4CSOx9TFsog1HYswAZMniFOYvLCRvzhzy5sxJKg8fPkx7dTWtbod62+5qPvr1rznyxBMAjAOiAns+81kCo0cnTKPi81kjRjh9I8F8GP9JZ4o5eiCh9fEqvPaIMxwJODcPjpnX0foYMxfyhvXRN2KM6YolC3OMrOJisubPJ2/+/HiZqhKpr6dt925e++ZfkRWBoRMnEKo9QOuOHUQOH07eid9P4LTTnOQxZjRZo2KJZIyTVMZfiG/q55y60Qgc3JWcQKp/RPxK62GTElof85xhSmz4dWP6lCULkxERIaukhKySEo4WOqeJzv6P/4ivjzY3EzpwgFDtAUK1tc50wHlt2rqV8AcHIRJJ2qd/2DAngcQSyZjRZI1aTOC8rxEoGYK/aS+y/1UneezdCG885m6YDaNmuS2PeU4iKTrdTl8Z4yFLFqZX+GId5JMmdbpew2HCBw92JJLa2nhiaduzh8aXX0Zbk8eJlLy8jkQy+osEivMJZDcR0PcJNO4ha8tqJPpfTuW8kuTWx+i59rhZY3qRJQvTJyQrK96f0RlVJXLkCKH9tYRq9xM+cCApqXR+qquEQEkxgaIcsvJCBN54i4C8RCAvQiA/QmDsBHzj57sPiZoPI6aD3/7JG3M87C/HDAgi4vSVFBeTO2N6p3U6PdXlnu5qrq0l/EEEIomtiSb8Ob8lkPc/TgIpEAKjR5E1fgqBqRUEZn0K/9ip6e8vMcZYsjAnj+M71VVLaN9e2mr20bjnMPrWYZzR8DcDK5EsJVCUS2BkCYFxEwlMmk5g3ISOq7pKS5Es+zMxxv4KgB9eNx6AK/s3DHOCMj7V9ed9hHZtIvzONkL7dhOqfZ/Q/r207t5HZN1LyRv5fARGjnQSR8KlwfGrukaNwpdro/Gawc+ShTllJJ3qmn1m8srmQ7D/j0T3/oHQri2E9u4kdKiFULOfUEsrofcP0bznTcINrR3jb7n8w4YldMSnXio8GlTtSi1z0rNkYQw4N/6VX4iv/EKyL4FsVTi0N3nsq/e3o+EQ4RY/IU4jFJhASEsIteYSagh1eVXX6QIRn7L3c5fjKyjAl5/fMRU4r/78/OR1ebHlPGd9QQGSm9v/z3g3pyxLFsZ0RgSGT3Km2dc4ZaFW5P3tBPZXEYglkcO/g2ygyA/zp6FjziEydAYh3xhCzVmE33+fd/7jh/iiUHz6OKJNTUSOHiVUW0u0qSk+oRmMsO/z4cvLS0k4efgLCpzkkppwEhPRMckor2ejD5tTniULYzIVyIGx850ppunDpNaH7PwVWW0PkQXkZg+B0XMomX2U1jY/Y2+cD/4g+APO5AuAP4j6stBQlEhblGhbiGhLiGhbmGhruzM1txFtaSPS0kq0ucWZmpqJNjYSbWqi/cN6JwnFEk/sWSdpSF6ek2zyUpNMcosmqSWU37HsL+goP+5np5iThiULY05Efgl8bJEzgdOfUV+dNPLuxDGNzqM9nr6l012IO2V8AW8QyA3CiEBH4nGTkEoQpYBoxE80kkU07Cca9hEJ+YiGhWhIiIYg2q7OFFKibWEibfVE6w8Sqg27icpJVhqKpA0HQAIBfPm5Tssn32395LkJpWAIvoLCjoSUkGSOSUYFBdbHM0BZsjCmN/l8UDrFmc50nvX1PxdNJSsryoWPvgCRdoiGnddIyJ3aIZowHyuPhnpcTyIhJNKOL16eUj9eNxZDSjyanBw0ipNgwj4iIXETjs8tc+fD4iajI055k49ogxAJCaGE7TScWTqcIIr4lLdmTnOezZyYNxILhC7Wx+Yl8aXLOpJa3tk+unofSXhJfaOU+sfU6fF6SS53X6c0txPNSb7owgueJgsRWQT8G87Dj+5T1R92UW8J8AQw331S3nhgF/C2W2WTqt7sZazGeCWqQnvID0PH9Hco6UUjCUnFSTz+SAh/pJ1ANCHBxJJNpkkuEkLDbe5ptFaizc3Oa0sbkZY253Rbi3ParfHdfaBKbukoJyZNeEns2knp50la1GMLVRO31+R9pRZp4psm71OT/5PwXonr9Zjtkqtrp/tOWnXMPtzlaPLHQEH64KHSniULEfEDK4GLcJ6pvVVEKlX1zZR6hcAtOHdJJdqjqinXNxpjPOXzOxO9/1AqwfnVmK5bfcuF0wC47Kebej2GwWid+32Vp6l3orwc52ABUK2qe1W1HXgUWNxJvTuAu4DWTtaZAahNcmkTuxHNmFOJl8liDPBewnKNWxYnInOAsar6TCfbTxCR10TkRRE5x8M4jTHGpOFln0VnlzPEz6yJiA+4F7ihk3oHgHGqWi8i84Bfich0VT2a9AYiNwE3AYwbN6634jbGGJPCy5ZFDTA2YbkMqE1YLgRmABtF5F3gbKBSRCpUtU1V6wFU9VVgDzAl9Q1UdZWqVqhqRWlpqUcfwxhjjJfJYitQLiITRCQIXAtUxlaqaoOqlqjqeFUdD2wCLnevhip1O8gRkYk4fTd7PYzVGGNMNzw7DaWqYRFZBmzAuQDiAVXdKSIrgCpVrexm83OBFSISBiLAzap6yKtYTc/YKL3GnHo8vc9CVdcB61LKbu+i7vkJ808CT3oZmzHGmMzZI8KMMcakZcN9ANNGDenvEIwxZkCzloUxxpi0LFkYY4xJy5KFMcaYtCxZGGOMScuShTHGmLQsWRhjjEnLLp01xmMPfiEIwGX9HIcxJ8JaFsYYY9KyZGGMMSYtOw1ljMf2BSb1dwjGnDBLFsDqRav7OwRjjOsHfzEVsD6eTPXV92WnoYwxxqRlycIYY0xaliyMMcak5WmyEJFFIvK2iFSLyPJu6i0RERWRipTycSLSKCLf8TJOY4wx3fMsWbjP0F4JXApMA74oItM6qVcI3AJs7mQ39wLPeRWjMcaYzHjZslgAVKvqXlVtBx4FFndS7w7gLqA1sVBErgD2Ajs9jNEYY0wGvEwWY4D3EpZr3LI4EZkDjFXVZ1LK84FbgR94GJ8xxpgMeZkspJMyja8U8eGcZvp2J/V+ANyrqo3dvoHITSJSJSJVdXV1JxSsMcaYrnl5U14NMDZhuQyoTVguBGYAG0UEYCRQKSKXA2cBS0TkLqAIiIpIq6r+Z+IbqOoqYBVARUWFYowxxhNeJoutQLmITAD2A9cC18VWqmoDUBJbFpGNwHdUtQo4J6H8H4HG1ERhjDGm73h2GkpVw8AyYAOwC3hcVXeKyAq39WCMMeYk4enYUKq6DliXUnZ7F3XP76L8H3s9MHNCpo0a0t8hnFTs+zKDgQ0kaHrMBl405tRjw30YY4xJy1oWxnjMWmJmMBDVwXHFaUVFhVZVVfV3GMYYc1IRkVdVtSJdPTsNZYwxJi1LFsYYY9KyZGGMMSYtSxbGGGPSsmRhjDEmLUsWxhhj0rJkYYwxJi1LFsYYY9KyZGGMMSatQXMHt4jUAftOYBclwIe9FE5vsrh6xuLqGYurZwZjXKeramm6SoMmWZwoEanK5Jb3vmZx9YzF1TMWV8+cynHZaShjjDFpWbIwxhiTliWLDqv6O4AuWFw9Y3H1jMXVM6dsXNZnYYwxJi1rWRhjjEnrlE0WInK1iOwUkaiIdHkVgYgsEpG3RaRaRJb3QVzDROTXIrLbfS3uol5ERLa5U6WH8XT7+UUkW0Qec9dvFpHxXsXSg5huEJG6hO/nK17H5L7vAyJyUER2dLFeROTf3bjfEJG5AySu80WkIeH7ur2P4horIr8VkV3u3+I3OqnT599ZhnH1+XcmIjkiskVEXnfj+kEndbz7e1TVU3ICpgIfAzYCFV3U8QN7gIlAEHgdmOZxXHcBy9355cCPuqjX2AffUdrPD/wv4Mfu/LXAYwMgphuA/+yHf1PnAnOBHV2svwx4DhDgbGDzAInrfOCZfvi+RgFz3flC4J1O/l/2+XeWYVx9/p2530GBOx8ANgNnp9Tx7O/xlG1ZqOouVX07TbUFQLWq7lXVduBRYLHHoS0GHnLnHwKu8Pj9upPJ50+Mdw3waRGRfo6pX6jqS8ChbqosBh5WxyagSERGDYC4+oWqHlDVP7rzHwG7gDEp1fr8O8swrj7nfgeN7mLAnVI7nT37ezxlk0WGxgDvJSzX4P0/mtNU9QA4/2iBEV3UyxGRKhHZJCJeJZRMPn+8jqqGgQZguEfxZBoTwOfd0xZrRGSsh/H0RH/8e8rUx93TG8+JyPS+fnP3dMkcnF/Lifr1O+smLuiH70xE/CKyDTgI/FpVu/y+evvvMas3djJQichvgJGdrPp7VV2byS46KTvhy8e6i6sHuxmnqrUiMhF4QUS2q+qeE40tRSaf35PvqBuZvN/TwC9UtU1Ebsb5pXWBhzFlqq+/q0z9EWfIh0YRuQz4FVDeV28uIgXAk8A3VfVo6upONumT7yxNXP3ynalqBDhTRIqAp0Rkhqom9kV59n0N6mShqhee4C5qgMRfpWVA7Qnus9u4ROQDERmlqgfc5vbBLvZR677uFZGNOL9+ejtZZPL5Y3VqRCQLGIq3pzzSxqSq9QmL/w38yMN4esKTf08nKvFAqKrrROS/RKREVT0fA0lEAjgH5EdU9ZedVOmX7yxdXP35nbnvecT9u18EJCYLz/4e7TRU97YC5SIyQUSCOB1Gnl155KoE/tKd/0vgmBaQiBSLSLY7XwJ8EnjTg1gy+fyJ8S4BXlC3d80jaWNKOad9Oc4554GgErjevcLnbKAhdsqxP4nIyNh5bRFZgHNcqO9+q155XwHuB3ap6r92Ua3Pv7NM4uqP70xESt0WBSKSC1wIvJVSzbu/x77szR9IE3AlThZuAz4ANrjlo4F1CfUuw7kaYg/O6Suv4xoOPA/sdl+HueUVwH3u/CeA7ThXAm0HvuxhPMd8fmAFcLk7nwM8AVQDW4CJffAdpYvpn4Gd7vfzW+CMPvo39QvgABBy/219GbgZuNldL8BKN+7tdHEVXj/EtSzh+9oEfKKP4lqIc4rkDWCbO13W399ZhnH1+XcGzAJec+PaAdzulvfJ36PdwW2MMSYtOw1ljDEmLUsWxhhj0rJkYYwxJi1LFsYYY9KyZGGMMSYtSxbG9ICINKav1e32a9y77hGRAhH5iYjscUcRfUlEzhKRoDs/qG+aNScXSxbG9BF3/CC/qu51i+7Dubu2XFWn44yWW6LOAInPA9f0S6DGdMKShTHHwb2j+G4R2SEi20XkGrfc5w79sFNEnhGRdSKyxN3sL3DvyBeRScBZwG2qGgVn6BZVfdat+yu3vjEDgjVzjTk+VwFnArOBEmCriLyEM/TKeGAmzojBu4AH3G0+iXM3NcB0YJs6A8N1Zgcw35PIjTkO1rIw5vgsxBnZNqKqHwAv4hzcFwJPqGpUVd/HGW4kZhRQl8nO3STSLiKFvRy3McfFkoUxx6erB8p096CZFpyxe8AZV2i2iHT3N5gNtB5HbMb0OksWxhyfl4Br3IfRlOI8unQL8ArOg5d8InIazuM3Y3YBkwHUefZIFfCDhNFLy0VksTs/HKhT1VBffSBjumPJwpjj8xTO6J+vAy8Af+uednoSZ2TXHcBPcJ6w1uBu8yzJyeMrOA/BqhaR7TjP3og9q+FTwDpvP4IxmbNRZ43pZSJSoM4T1IbjtDY+qarvu88g+K273FXHdmwfvwS+p+mfE29Mn7CroYzpfc+4D6kJAne4LQ5UtUVE/gHnOcl/7mpj96FOv7JEYQYSa1kYY4xJy/osjDHGpGXJwhhjTFqWLIwxxqRlycIYY0xaliyMMcakZcnCGGNMWv8fBVygby15aLoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xad1c940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid.cv_results_['mean_test_score']\n",
    "test_stds = grid.cv_results_['std_test_score']\n",
    "train_means = grid.cv_results_['mean_train_score']\n",
    "train_stds = grid.cv_results_['std_train_score']\n",
    "\n",
    "# plot results\n",
    "n_Cs = len(Cs)\n",
    "number_penaltys = len(penaltys)\n",
    "test_scores = np.array(test_means).reshape(n_Cs, number_penaltys)\n",
    "train_scores = np.array(train_means).reshape(n_Cs, number_penaltys)\n",
    "test_stds = np.array(test_stds).reshape(n_Cs, number_penaltys)\n",
    "train_stds = np.array(train_stds).reshape(n_Cs, number_penaltys)\n",
    "\n",
    "x_axis = np.log10(Cs)\n",
    "for i, value in enumerate(penaltys):\n",
    "    plt.errorbar(x_axis, -test_scores[:,i], yerr = test_stds[:,i], label = penaltys[i] + 'Test')\n",
    "    plt.errorbar(x_axis, -train_scores[:,i], yerr = train_stds[:,i], label = penaltys[i] + 'Train')\n",
    "    \n",
    "plt.legend()\n",
    "plt.xlabel('log(C)')\n",
    "plt.ylabel('logloss')\n",
    "plt.savefig('LogisticGridSearchCV_C.png')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上图给出了L1正则和L2正则下、不同正则参数C对应的模型在训练集上及测试集上的logloss。可以看出在训练集上C越大（正则越少）的模型性能越好；但在测试集上随着C增大，logloss先减小后又增大？？？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 保存模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "pickle.dump(grid.best_estimator_, open('Diabetes_L1_org.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
